existential risk
How a Contractor's Claude Agent Got Into the Pentagon
This week on, we also discuss Anthropic's IPO filing and the volunteers gearing up to counter ICE at the midterms. This week on, Brian Barrett, Leah Feiger, and Zoë Schiffer dive into our scoop on how a defense contractor connected Claude Code to a secure Department of Defense system. We also discuss the green, yellow, and red flags we see in Anthropic's IPO filing--and why it's a test not just for the AI industry, but for the entire US economy--and how volunteers nationwide are preparing for ICE agents potentially showing up at polling sites in the midterms. Write to us at [email protected] . You can always listen to this week's podcast through the audio player on this page, but if you want to subscribe for free to get every episode, here's how: If you're on an iPhone or iPad, open the app called Podcasts, or just tap this link . There is a lot going on in the hundreds of pages that the company submitted as part of its S-1 filing, from staggering earnings and losses, to warnings of AI's existential threat. We'll discuss our key takeaways and why this IPO is kind of high stakes, not just for the AI industry, but honestly the entire US economy. Massive groups of volunteers all over the country are already getting ready to protect their communities from the potential that ICE officers show up at the polls. We'll get into how grassroots efforts like these are a reflection of a deeply uncertain election cycle. This has been highly anticipated for a bunch of reasons. Anthropic is expected to raise more money in its IPO than any company in history, and it would be the first major US company focused solely on building AI software to go public. To give you a sense of the scale that we're talking about, the company's valuation went from $183 billion a little over a year ago to $965 billion in its most recent round, and some expect that it could be worth close to $2 trillion once it starts trading.
'Extraordinary potential' and 'catastrophic' risk: Anthropic's 5 biggest IPO head-turners
The Dell Laptop Buying Guide UK: Which Dell Laptop Should You Buy? When you purchase through links in our articles, we may earn a small commission. Its models could transform work as we know it, cure disease, and/or destroy humanity. No, Anthropic's IPO filing is not like the others. I don't make a habit of perusing the paperwork from tech companies -- or any companies, really-looking to go public.
The Anthropic IPO Is Also a Doomsday Primer
Anthropic thinks "closing the loop" is the best path to AI triumph--and a recipe for "existential" threats to humanity. Anthropic has officially filed the prospectus for its initial public offering, which means the AI lab has put all kinds of formerly private information on display. While the prospectus contains the eye-popping revenue--and even larger expenditures--you might expect, the biggest section of the filing by far is given over to "risk factors," which are all the things that could squash Anthropic's big plans. While much has been made about the extent to which AI represents an existential threat to humankind, the specific mechanisms by which it could bring about that destruction are largely opaque. If nothing else, the 94 pages that Anthropic's prospectus devotes to risk factors make clear both where it sees the danger coming from, and the ways in which its researchers are running toward it in order to win the AI arms race.
Timnit Gebru Believes There Is No 'Existential Threat' From AI
Timnit Gebru Believes There Is No'Existential Threat' From AI One of AI's fiercest critics believes the doom talk is about founders making money, not saving humanity. As the industry booms, an entirely new vernacular has emerged in the debates over how consequential artificial intelligence really is. Within that new parlance, two phrases have become focal points: stochastic parrots and existential risk . And at the center of is one prominent AI researcher who has never stood down from a fight, Timnit Gebru . Gebru came into the public eye several years ago after sparring with Google over a research paper she coauthored that called out biases in the company's AI, saying that LLMs basically parroted their training data and risked perpetuating biased viewpoints. The contested paper resulted in Gebru's departure from the company, and led her to found an institute that investigates harms perpetuated by technology and supports the creation of unbiased tech tools. She has also authored a new book,, expected to ship early next year. More recently, Gebru has spoken out against a faction of the industry that believes AI is so powerful it could destroy humanity. In turn, some members of that community, including an Anthropic cofounder, have alleged that Gebru's earlier research about stochastic parrots is no longer relevant, and that AI does have the ability to "think" or reason. The AI debate is no longer just about the technology itself, but about ideological groups and strategic narratives, and Gebru believes these narratives are a "harmful distraction" from the real issues with AI. I recently spoke with Gebru about what she believes these arguments are distracting from, and also got her response to critiques that her earlier research underestimates the AI of today.
Why China Isn't Getting Existential About A.I.
China's fears about the technology are increasingly distinct from those of the U.S. What does that mean for international coöperation? On Sunday, China's Minister of State Security published an article in a state-run magazine that warned about artificial intelligence posing a threat to the rule of the Chinese Communist Party (C.C.P.). This coincided with increasing concern among Americans that A.I. could pose existential risks to humanity, and rising calls for Washington to regulate it. Many of those pushing for regulation work at leading A.I. companies such as Anthropic and OpenAI, and they have received support from a broad ideological coalition on Capitol Hill. But there remains little appetite for regulation in the White House; President Trump has said that the United States must accelerate A.I. development to outpace China. I recently spoke by phone with Kyle Chan, a fellow at the Brookings Institution who focusses on Chinese technology policy.
How AI firm Anthropic wound up in the Pentagon's crosshairs
This week has brought more chaos in the feud between the Pentagon and Anthropic. This week has brought more chaos in the feud between the Pentagon and Anthropic. How AI firm Anthropic wound up in the Pentagon's crosshairs U ntil recently, Anthropic was one of the quieter names in the artificial intelligence boom. Despite being valued at about $350bn, it rarely generated the flashy headlines or public backlash associated with Sam Altman's OpenAI or Elon Musk's xAI. Its CEO and co-founder Dario Amodei was an industry fixture but hardly a household name outside of Silicon Valley, and its chatbot Claude lagged in popularity behind ChatGPT.
AI Consciousness and Existential Risk
In AI, the existential risk denotes the hypothetical threat posed by an artificial system that would possess both the capability and the objective, either directly or indirectly, to eradicate humanity. This issue is gaining prominence in scientific debate due to recent technical advancements and increased media coverage. In parallel, AI progress has sparked speculation and studies about the potential emergence of artificial consciousness. The two questions, AI consciousness and existential risk, are sometimes conflated, as if the former entailed the latter. Here, I explain that this view stems from a common confusion between consciousness and intelligence. Yet these two properties are empirically and theoretically distinct. Arguably, while intelligence is a direct predictor of an AI system's existential threat, consciousness is not. There are, however, certain incidental scenarios in which consciousness could influence existential risk, in either direction. Consciousness could be viewed as a means towards AI alignment, thereby lowering existential risk; or, it could be a precondition for reaching certain capabilities or levels of intelligence, and thus positively related to existential risk. Recognizing these distinctions can help AI safety researchers and public policymakers focus on the most pressing issues.
AI Safety Should Prioritize the Future of Work
Hazra, Sanchaita, Majumder, Bodhisattwa Prasad, Chakrabarty, Tuhin
Current efforts in AI safety prioritize filtering harmful content, preventing manipulation of human behavior, and eliminating existential risks in cybersecurity or biosecurity. While pressing, this narrow focus overlooks critical human-centric considerations that shape the long-term trajectory of a society. In this position paper, we identify the risks of overlooking the impact of AI on the future of work and recommend comprehensive transition support towards the evolution of meaningful labor with human agency. Through the lens of economic theories, we highlight the intertemporal impacts of AI on human livelihood and the structural changes in labor markets that exacerbate income inequality. Additionally, the closed-source approach of major stakeholders in AI development resembles rent-seeking behavior through exploiting resources, breeding mediocrity in creative labor, and monopolizing innovation. To address this, we argue in favor of a robust international copyright anatomy supported by implementing collective licensing that ensures fair compensation mechanisms for using data to train AI models. We strongly recommend a pro-worker framework of global AI governance to enhance shared prosperity and economic justice while reducing technical debt.